Systems and methods for colorimetric and spectral material estimation

ABSTRACT

Systems, devices, and methods for generating signatures for an image obtain an image, estimate a spectral image of the obtained image, calculate one or both of a detection component of the image and a residual component of the image, wherein the detection component of the image is based on the spectral image, a spectral-power distribution for a specific illuminant, and spectral sensitivities of a detector, and wherein calculating the residual component of the image is based on the spectral image, the spectral power distribution for the specific illuminant, and the spectral sensitivities of the detector or, alternatively, based on the spectral image and the calculated detection component, and generate an image signature based on one or both of the detection component and the residual component.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.61/669,529, which was filed on Jul. 9, 2012, to U.S. ProvisionalApplication No. 61/736,130, which was filed on Dec. 12, 2012, and toU.S. Provisional Application No. 61/793,879, which was filed on Mar. 15,2013.

BACKGROUND

1. Technical Field

The present disclosure generally relates to colorimetric and spectralmaterial estimation.

2. Background

Human beings, as well as digital imaging systems that mimic humanvision, capture chromatic signals that can be represented bydevice-dependent red-green-blue (“RGB”) values or device-independenttristimulus values (“XYZ”) under specific illumination conditions for astandard observer, resulting in a set of parameters that indicatelightness, hue, and chroma. These illuminant-dependent,observer-dependent color encodings estimate how the human visual systemperceives color, as well as how different areas of a scene are encodedas an image by a digital imaging system. Although trichromatic systemsgive good estimates of the appearances of objects in a scene, theysuffer from metamerism, which is a phenomenon in which two differentphysical stimuli result in identical sets of tristimulus values whenintegrated with either illuminant spectral-power distribution, detectorspectral sensitivities, or both.

SUMMARY

In one embodiment, a method comprises obtaining an image; estimating aspectral image of the obtained image; calculating one or both of adetection component of the image and a residual component of the image,wherein the detection component of the image is based on the spectralimage, a spectral-power distribution for a specific illuminant, andspectral sensitivities of a detector, and wherein calculating theresidual component of the image is based on the spectral image, thespectral power distribution for the specific illuminant, and thespectral sensitivities of the detector or, alternatively, based on thespectral image and the calculated detection component; and generating animage signature based on one or both of the detection component and theresidual component.

In one embodiment, a device for generating a spectral signaturecomprises one or more computer-readable media and one or more processorsconfigured to cause the device to perform operations including obtainingan image, generating a spectral reflectance estimate of the image, andcalculating one or both of a detection component and a residualcomponent, wherein the detection component is based on the spectralreflectance estimate of the image, a spectral-power distribution for aspecific illuminant, and detector spectral sensitivities, and whereinthe residual component is based on the spectral reflectance estimate ofthe image and the calculated detection component or, alternatively,based on the spectral reflectance estimate of the image, the spectralpower distribution for the specific illuminant, and the detectorspectral sensitivities.

In one embodiment, one or more computer-readable media storeinstructions that, when executed by one or more computing devices, causethe computing device to perform operations comprising obtaining animage; generating a spectral image of the obtained image; and generatingone or more of a detection component and a residual component, whereinthe detection component is based on the spectral image, a spectral-powerdistribution for a specific illuminant, and detector spectralsensitivities, and wherein the residual component is based on thespectral image and the calculated detection component or is based on thespectral reflectance image, the spectral power distribution for thespecific illuminant, and the detector spectral sensitivities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example embodiment of the operations that areperformed by a system for material estimation.

FIG. 2 illustrates an example embodiment a method for generatingcomponent coefficients.

FIG. 3 illustrates an example embodiment of a method for generatingcomponent signatures.

FIG. 4 illustrates an example embodiment of a method for generatingcomponent signatures.

FIG. 5 illustrates an example embodiment of the operations that areperformed by a system for material identification.

FIG. 6 illustrates the spectral reflectance of certain objects.

FIG. 7 illustrates an example embodiment of a method for identifying amaterial.

FIG. 8 illustrates an example embodiment of a method for identifying amaterial.

FIG. 9 illustrates an example embodiment of a system for identifying amaterial.

FIG. 10A illustrates an example embodiment of a system for identifying amaterial.

FIG. 10B illustrates an example embodiment of a system for identifying amaterial.

FIG. 11 illustrates an example embodiment of a candidate materialdataset.

FIG. 12 illustrates an example of spectral reflectances of materials ina candidate material dataset.

FIG. 13 illustrates examples of spectral reflectance correlationsbetween a material and materials in a candidate material dataset.

FIG. 14 illustrates examples of spectral reflectance correlationsbetween a material and materials in a candidate material dataset.

FIG. 15 illustrates examples of spectral reflectance correlationsbetween a material and materials in a candidate material dataset.

FIG. 16 shows the discrimination success rates for examples of overallspectral correlations and residual component correlations.

FIG. 17 illustrates an example embodiment of a candidate materialdataset.

FIG. 18 illustrates an example of spectral reflectances of materials ina candidate material dataset.

FIG. 19A shows the average overall spectral correlations for 88 naturalsamples and 104 man-made samples for different numbers of channels.

FIG. 19B shows the average correlations of material samples that werecalculated based on fundamental components.

FIG. 19C shows the average correlations of material samples that werecalculated based on metameric black components.

FIG. 20A shows a comparison between overall spectral correlation resultsand metameric black correlation results for 88 natural samples.

FIG. 20B shows the improvement of metameric black correlation resultsover overall spectral correlation results for 88 natural samples.

FIG. 21A shows a comparison between the results of overall spectralcorrelation and metameric black correlation for 104 man-made materialsamples.

FIG. 21B shows the improvement of metameric black correlation resultsover overall spectral correlation results for 104 man-made materialsamples.

FIG. 22 shows the discrimination success rates for examples of overallspectral correlations and residual component correlations.

FIG. 23A illustrates a graphic representation of a transformation matrix[T].

FIG. 23B shows an example of detection components.

FIG. 23C shows an example of residual components.

DESCRIPTION

The following disclosure describes certain explanatory embodiments.Other embodiments may include alternatives, equivalents, andmodifications. Additionally, the explanatory embodiments may includeseveral novel features, and a particular feature may not be essential tosome embodiments of the devices, systems, and methods described herein.

FIG. 1 illustrates an example embodiment of the operations that areperformed by a system for material estimation. The system includes oneor more computing devices (e.g., desktops, laptops, tablets, servers,phones, PDAs), although only certain computing-device components areshown in FIG. 1 in order to emphasize the operations in thisillustration.

The system includes a spectral-estimation module 120, which receives oneor more images 101 and generates one or more corresponding spectralimages 102 from the one or more images 101. An image 101 is anelectronic representation of a scene that is captured by convertingphotons coming from the scene into electrons that produce spatial arraysof a limited number of channels that correspond to captured bands oflight wavelengths. Also, although an image is described, the system, aswell as the other systems, devices, and methods described herein, canoperate on one or more regions of an image. Modules includecomputer-readable data or computer-executable instructions, and may beimplemented in software (e.g., Assembly, C, C++, C#, Java, BASIC, Perl,Visual Basic), hardware (e.g., customized circuitry), or a combinationof software and hardware. In some embodiments, the system includesadditional or fewer modules, the modules are combined into fewermodules, or the modules are divided into more modules. Though thecomputing device or computing devices that execute a module actuallyperform the operations, for purposes of description a module may bedescribed as performing one or more operations.

A spectral image 102 defines values for physical properties that aremeasured across a range of wavelengths, and thus the values describe amaterial's physical property that is sampled across a range ofwavelengths. Examples of physical properties include spectralreflectance, spectral radiance, spectral scattering, and spectraltransmittance. Accordingly, in some embodiments, a spectral image 102defines spectral reflectance values, which define the ratio of the totalamount of light radiation reflected by a surface to the total amount ofradiation incident on the surface. Therefore, the spectral image 102defines the values (e.g., presents a map of the values) of reflectanceproperties of the material or materials in the image 101. Moreover, someembodiments directly capture the spectral image 102, and thus theobtaining of the image 101 and the spectral estimation module 120 areomitted from these embodiments.

In FIG. 1, a weight-calculation module 125 obtains detector spectralsensitivities information 103, which describes the spectralsensitivities of a detector (e.g., a human observer, an electronic imagesensor), obtains illuminant spectral-power distribution information 104,and generates weights W 105 based on the received information. Adetection-component module 130 obtains the weights W 105 and the one ormore spectral images 102 and generates a detection component D 106 basedon them. A residual-component module 135 obtains the one or morespectral images 102, as well as one or more of the detection component D106 and the weights W 105. Also, the residual-component module 135generates a residual component Res 107 based on the obtained one or morespectral images 102 and based on the detection component D 106 or theweights W 105.

A detection signature module 140 obtains the detection component D 106and a priori basis functions Ed 112 and generates a detection signatureAd 108 for the image 101 based on them. A residual signature module 145obtains the residual component Res 107 and a priori basis functions Er113 and generates a residual signature Ar 109 for the image 101 based onthem. Also, some embodiments use multiple illuminants and detectorspectral sensitivities to create several sets of detection component andresidual component pairs, which may be used for material identification.Also, one or both of the detection signature Ad 108 and the residualsignature Ar 109 may be referred to as an image signature of the image101.

A material identification module 150 obtains the detection signature Ad108, the residual signature Ar 109, and a priori material signatures 114and uses them to generate a material identification 111, for example anestimate of the materials in the one or more images 101.

For example, some embodiments include an a priori table of materialsignatures created based on representative materials, and thecorrelation between the joint colorimetric and spectral materialsignature of a testing material could be used to estimate a probabilityvalue for each material in the table. The correlation may be representedas a two-dimensional map in which one axis represents a distance in thedetection-signature space between tested samples and candidatematerials, and the other axis represents the residual-signature spectraldifference between tested samples and candidate materials. The overallmetric could be a weighted metric between the distance in detectionspace and the residual spectral difference.

Thus, the system enables an examination of materials without restrictingthe examination to parameters calculated based on human visual systemproperties, but at same time includes an examination that is based onthe human visual system.

FIG. 2 illustrates an example embodiment of the operations that areperformed by a system for material estimation. The system receives animage or, alternatively, a region of an image, collectively referred toas “image Im 201,” with dimensions c by x by y, where x is the number ofrows, y is the number of columns, and c represents the number ofchannels. The size of the image Im 201 is defined by the dimensions xand y. For example, the image Im 201 could be a 1,000 by 1000 pixelimage or a 1 by 1 pixel image. If the image Im 201 is a 1 by 1 pixelimage, then the dimensions simplify to c by 1. The image Im 201 is usedby the spectral estimation module 220 to generate a spectral image Spec202 with dimensions m by x by y, where m is the number of reflectancesamples for different wavelengths of light. Systems and methods forestimating a spectral image Spec 202 from a multi-channel image Im 201are described in U.S. Patent Publication No. 20120213407, which ishereby incorporated by reference.

The effect of the combination of illuminant spectral-power distributionand spectral sensitivities of an image acquisition system on thespectral reflectances of the original scene can be mathematicallydescribed as a projection of reflectances in the sensing representationspace, represented by the channels of the image Im 201 that are capturedby a camera. In mathematical terms, the sensing is a transformation [T]215 (e.g., a projection) that may be calculated as the term-by-termproduct of the detector spectral sensitivities O 203 (e.g., the spectralsensitivities of the imaging system) and the illuminant spectral-powerdistribution S 204, for example according to [T]=W*(W′*W)⁻¹*W′, wherethe weight matrix W 205 is an m by c matrix of weights calculated as aterm-by-term product between the normalized illuminant spectral-powerdistribution S 204 (e.g., of a selected illuminant), with dimensions mby 1, and the detector spectral sensitivities O 203, with dimensions mby c; where W′ denotes the transposed matrix of the weight matrix W 205;and where the superscript −1 denotes an inverse matrix. Thetransformation [T] may be defined by a transformation matrix [T].

Thus, in the embodiment illustrated by FIG. 2, a weight matrixcalculation module 225 obtains the detector spectral sensitivities O 203and the illuminant spectral-power distribution S 204 and generates aweight matrix W 205 based on them. To generate the weight matrix W 205,for each channel the term for each of the detector channel sensitivitiesshould be multiplied by the corresponding term in the spectral-powerdistribution for the same wavelength. For example, the weight matrix W205 may be calculated according toW(i,j)=S(j)*O(j,i),for i=1:c and j=1:m.

A transformation calculation module 255 then generates a transformation[T] 215 based on the weight matrix W 205, where [T] is an m by m matrix.When applied to the spectral image Spec 202, the transformation [T] 215projects reflectance space into image acquisition space (e.g., the m byx by y space), producing the detection component D 206. The detectioncomponent calculation module 230 calculates the detection component D206 of the captured channels of the image Im 201 according toD=[T]*Spec. The detection component D 206, with dimensions m by x by y,carries the information captured by an image acquisition system under aspecific illumination. Therefore, the detection component D 206 is thespectral component of the spectral image that is related to thecombination of detector spectral sensitivities and the spectral powerdistribution of a specified illuminant. Note that the detector spectralsensitivities O 203 of the detector used in the calculation do not haveto correspond to the detector that actually captured the image Im 201,but may correspond to a detector (e.g., an imaging sensor) that isdetermined a priori and is used to build a database of materialsignatures.

The rejected or ignored residual component Res 207, with dimensions m byx by y, carries no information for the imaging sensor, and the nullsensor capture termed is called residual. Therefore, the residualcomponent Res 207 is the spectral component of the spectral image thatdoes not contribute to the detection component. The residual componentcalculation module 235 can calculate the residual component Res 107based on the detection component D 206 and the spectral image Spec 202or based on the transformation [T] 215 (or the weight matrix W 205) andthe spectral image Spec 202. For example, the residual componentcalculation module 235 can calculate the residual component Res 207according to Res=(Id−[T])*Spec, where Id is an m by m identity matrix.Also, the residual component calculation module 235 can calculate theresidual component Res 207 according to Res=Spec−D. Thus, thecalculation of the residual component Res 207 does not require thecalculation of the detection component D 206, although the calculationof the residual component Res 207 can use the detection component D 206.The residual component Res 207 can be correlated to the spectral curveshapes of respective materials that are not captured by the sensor. Insome embodiments, the residual component Res 207 is a metameric black,for example when human-visual-system color-matching functions areconsidered as detector spectral sensitivities.

Also, the detection component decomposition module 260 decomposes thedetection component D 206 based on the detection component D 206 anddetection component basis functions Ed 212 to generate the detectioncomponent coefficients Ad 216, for example according to D=Ad*Ed. Theresidual component decomposition module 265 decomposes the residualcomponent Res 207 based on the residual component Res 207 and residualcomponent basis functions Er 213 to generate the residual componentcoefficients Ar 217, for example according to Res=Ar*Er. Additionally,the basis functions could be calculated, for example, by eigenvectoranalysis, independent component analysis, etc. Furthermore, one or moreof the coefficients Ad 216 and Ar 217 may give a material signature, orthe material signature may be otherwise based on Ad 216 and Ar 217.Moreover, considering that surface reflectances may fall within a linearmodel composed of band-limited functions with a small number ofparameters, the spectral component(s) may be decomposed into a smallnumber of components without losing the capability to reconstruct theoriginal spectra.

Additionally, in some embodiments, the operations do not includecalculating the residual component Res 207 and the residual componentcoefficients Ar 217. Furthermore, in some embodiments the operations donot include calculating the detection component D 206 and the detectioncomponent coefficients Ad 216.

As a further illustration of the operation of FIG. 2, in an exampleembodiment the spectral image Spec 202 is a reflectance image Ref 202,which defines spectral reflectance values. Thus, Spec=Ref in thisexample. Accordingly, the detection component D 206 may be calculatedaccording to D=[T]*Spec, and the residual component Res 207 may becalculated according to Res=(Id−[T])*Spec or Res=Spec−D.

Also, as noted previously, the operations described above could beperformed on a pixel basis, for example pixel-by-pixel, in which thevariable x corresponding to the row number ranges from x1 to xi, wherex1 is the first row and xi is the last row of the image Im 201; and foreach row x, the variable y corresponding to the number of the columnranges from y1 to yj, where y1 is the first column and yj is the lastcolumn of the image Im 201. The image Im 201 (c,x,y), the spectral imageSpec 202 (m,x,y,), the detection component D 206 (m,x,y), and theresidual component Res 207 (m,x,y) are general representations of,respectively, a captured image (or region of an image), a spectralimage, a detection component, and a residual component. In a case inwhich x=1 and y=1, then there is only one pixel or is only one pixelsample of an image or of a measurement. A sample taken from a set ofsamples, represented by dimensions x and y, could be obtained bycropping regions-of-interest in the image, by creating an array ofnon-contiguous samples in an image, or by performing a calculation on acertain number of images (e.g., a statistical measure, such as mean ormedian over a sampled number of pixels).

FIG. 3 illustrates an example embodiment of a method for generatingcomponent signatures. The blocks of this method and the other methodsdescribed herein may be performed by one or more computing devices, forexample the systems and devices described herein. Also, otherembodiments of this method and the other methods described herein mayomit blocks, add blocks, change the order of the blocks, combine blocks,or divide blocks into more blocks.

First, in block 300, an image (or a region of an image) is obtained. Forexample, an image may be received by a computing device. Also, the imagemay be a multi-spectral image. By capturing several channels withoptical filters that have different spectral transmittances, someimage-capturing devices sample the spectra in a manner that allows theestimation of the spectral reflectance of objects (e.g., Lambertianobjects) in a scene with a relatively high accuracy (e.g., by using thesmooth nature of spectral reflectances of objects for datainterpolation) based on the captured image.

Next, in block 310, a spectral image is generated based on the image.The flow then moves to block 320, where a detection component iscalculated based on the spectral image, one or more detector spectralsensitivities, and illuminant spectral-power distribution. Calculatingthe detection component may include generating a weight matrix and atransformation. The flow then proceeds to block 330, where a residualcomponent is calculated based on the spectral image, one or moredetector spectral sensitivities, and illuminant spectral-powerdistribution. Calculating the residual component may include generatinga weight matrix and a transformation, for example in embodiments thatomit block 320. Also, in some embodiments the residual component iscalculated based on the spectral image and the detection component.Next, in block 340, a detection-component signature is generated basedon the detection component and one or more detection-component basisfunctions. Finally, in block 350, a residual-component signature isgenerated based on the residual component and one or more residualcomponent basis functions. Also, some embodiments generate a materialsignature based on the detection-component signature and theresidual-component signature.

Other embodiments of the method may not calculate a detection component,and thus may omit block 340. Some of these other embodiments may includeblock 320 and calculate the residual component based on the detectioncomponent, but some of these other embodiments may also omit block 320.Additionally, other embodiments of the method may not calculate aresidual component, and thus may omit blocks 330 and 350.

FIG. 4 illustrates an example embodiment of a method for generatingcomponent signatures. The embodiment illustrated by FIG. 4 calculates ametameric black component for the residual component and calculates afundamental component for the detection component, although thedecomposition into detection and residual components is a more generalcase than the decomposition into the fundamental component and themetameric black component, and the general case can account for morediverse image acquisition sensitivities.

For example, the Wyszecki fundamental component is a special case ofcomponent detection for the human visual system. The Wyszeckifundamental component N is calculated according to N=[R]*Spec, and themetameric black component B is calculated according to B=(I−[R])*Spec orB=Spec−N, where I is an m by m identity matrix. The transformation [R]is calculated according to [R]=A*(A′*A)⁻¹*A′, where A is an m by 3matrix of tristimulus weights that were calculated as a product betweenthe normalized spectral-power distribution S of a selected illuminantand the detector spectral sensitivities O, which are the color matchingfunctions of a human observer (which are an embodiment of detectorspectral sensitivities); where A′ denotes the transpose matrix of A; andwhere the superscript⁻¹ denotes an inverse matrix. The fundamentalcomponent N, with dimensions m by x by y, is the portion that istransmitted by the human visual system and that carries all informationnecessary for human color sensation. The metameric black component B,with dimensions m by x by y, is portion that is rejected or ignored bythe human visual system and that carries no information for human colorsensation, but instead evokes the null color “sensation” termedmetameric black.

In FIG. 4, the flow starts in block 400, where an image is obtained(e.g., received, captured). Next, in block 410, a spectral image isgenerated based on the obtained image. The flow then proceeds to block420, where a fundamental component N is calculated based on the spectralimage, detector spectral sensitivities (e.g., the color matchingfunctions of a human observer), and illuminant spectral-powerdistribution. For example, the fundamental component N may be calculatedaccording to N=[R]*Spec.

The flow then moves to block 430, where a detection component signatureis generated based on the fundamental component N. For example, thefundamental component N may be decomposed into its basis function En andtheir respective coefficients An (e.g., using eigenvector analysis,using independent component analysis), and the detection componentcoefficients An may be used as the detection component signature.

Next, in block 440, a metameric black component B is calculated based onthe spectral image, detector spectral sensitivities (e.g., the colormatching functions of a human observer), and illuminant spectral-powerdistribution. Also, in some embodiments the metameric black component Bis calculated based on the spectral image and the fundamental componentN.

Finally, in block 450 a residual component signature is generated basedon the metameric black component B. For example, the metameric blackcomponent B may be decomposed into its basis function Eb and theirrespective coefficients Ab (e.g., using eigenvector analysis, usingindependent component analysis), and the metameric black componentcoefficients Ab may be used as the metameric black component signature.

FIG. 5 illustrates an example embodiment of the operations that areperformed by a system for material identification. The system includesone or more cameras (or other image sensors) and one or more computingdevices, although only certain components are shown in FIG. 5 in orderto emphasize the operations in this illustration. A camera 595 capturesan image Im 501 of an object 597. The camera 595 may have tunablesensitivities, may be configured to capture light fields, and may beconfigured to capture multi-channel images. The image Im 501 is receivedby an image-signature generation module 560, which may be a component ofthe camera 595 or a component of a separate device. The image-signaturegeneration module 560 generates an image signature 561 based on theimage Im 501. The image signature 561 may include one or more of adetection signature and a residual signature, or may include a combinedsignature that was generated from the detection signature and theresidual signature. The image signature 561 is sent to amaterial-identification module 550.

The material-identification module 550 also obtains material signature514 from a database of material signatures 501, which stores previouslygenerated material signatures 514, for example material signatures 514that were produced by experimental analysis. Based on the imagesignature 561 and the material signatures 514, thematerial-identification module 550 generates a material identification511 that identifies one or more materials in the image, which correspondto the materials of the object 597.

For example, to generate the material identification 511, in someembodiments the material-identification module 550 compares imagesignatures 561 with pre-determined material signatures 514. An imagesignature 561 and a pre-determined material signature 514 may eachinclude a respective multi-dimensional representation of theirrespective materials, and the material-identification module 550 maycalculate correlations between the image signature 561 and thepre-determined material signatures 514. The material-identificationmodule 550 may separately evaluate detection component signatures andresidual component signatures, or may combine them into a singlesignature. For example, the material-identification module 550 maycompare the detection component signature of a test object with thedetection component signature of a reference object (e.g., a candidateobject whose information is stored in the database of materialsignatures 501) and separately compare the residual component signatureof the test object with the residual signature of the reference object.Or the material-identification module 550 may use a detection componentsignature and a residual component signature that are combined into asingle material signature and compare the single signature of a testobject with the single signature of a reference object.

To calculate correlations, some embodiments calculate a probabilityfunction using a very simple equation: P=(C+1)/2, where C is the Pearsoncorrelation (i.e., a covariance of the samples divided by the product ofthe standard deviations for each sample). The probability function maybe used for both colorimetric components and spectral components, whichmay give more flexibility to consider both color visual perception andmaterial spectral ground-truth matching by decoupling colorimetric andspectral components. Separating the color component from the objectspectral component may improve discrimination capabilities through useof the residual spectral component because the separation may eliminatecolor aspects in man-made objects that have a large weight in spectralcolor signatures when only the overall spectra is used. Also, systemsand methods may benefit from using the color component. Therefore, someembodiments use a detection probability function Pn=(Cn+1)/2 that isdefined based on the detection component Cn correlation, use a residualprobability function Pr=(Cr+1)/2 is defined based on the residualcomponent Cr correlation, or use both.

For material property discrimination, correlation compares the overallsimilarity in shapes between two spectral reflectances. Thus, a metricderived based on correlation may be more effective than theroot-mean-square (“rms”) (also referred to as the “quadratic-meanserror”) technique that is sometimes used in spectral curve comparisonsbetween reference spectra and estimated spectra.

For example, FIG. 6 illustrates the spectral reflectance of certainobjects. Graph 1 illustrates the spectral reflectance of a test orange(continuous line) and the spectral reflectance of a reference orange(dotted line). The spectral reflectance of the test orange has a verysimilar shape to the spectral reflectance of the reference orange. Aprobability metric generated using correlation gives a match of 91%between these two spectral curves. However, the rms spectral errorbetween these curves is 21%, due to the magnitude shift. Graph 2illustrates the spectral reflectance of the test orange (continuousline) and the spectral reflectance of a reference banana (dotted line).If the rms spectral error is used as the criterion for matchingmaterials, then the smaller rms spectral error (8%) relative to the testorange reflectance indicates that the material is a “banana”. But,although the spectral curves of the test orange and the reference bananaare very close to each other (which results in a low spectral rmserror), the overall shapes of the spectral curves are very differentfrom each other.

Additionally, when performing the comparison, the detection signature Adand the residual signature Ar may be weighted. If the visual appearanceis more important in the implementation application, the detectionsignature Ad can be given a larger weight than the residual signatureAr. If the ground-truth information is more important, the residualsignature Ar can be assigned a larger weight than the detectionsignature Ad. In certain applications, both the spectral curve shape andthe visual appearance of the object are important (e.g., melanomadetection) and the detection signature Ad and the residual signature Arare considered as a whole (e.g., equally weighted).

FIG. 7 illustrates an example embodiment of a method for identifying amaterial. The flow starts in block 700, where an image (or a region ofan image) is obtained. Next, in block 710, one or more materialsignatures, which may each include one or both of a detection componentsignature and a residual component signature, are generated based on theimage and one or more a priori basis functions.

The flow then moves to block 720, where the one or more materialsignatures are compared to the respective one or more materialsignatures of candidate materials, for example by calculatingcorrelations between the one or more material signatures and therespective one or more material signatures of the candidate materials.Finally, in block 730, a candidate material is selected based on thecomparisons.

FIG. 8 illustrates an example embodiment of a method for identifying amaterial. The flow starts in block 800, where an image is obtained.Next, in block 805, the count i is initialized to 0. The flow then movesto block 810, where it is determined if another pair of a detectionsignature and a residual signature is to be generated. If there are Igroups of spectral sensitivities and illuminant spectral-powerdistributions for which signatures are to be calculated, then, if i<I(block 810=yes) the flow proceeds to block 815. In block 815, adetection signature and a residual signature are generated based on theimage and the current group of detector spectral sensitivities andilluminant spectral-power distributions. The flow then moves to block820, where the count i is incremented, and then the flow returns toblock 810.

If in block 810 it is determined that another pair of a detectionsignature and a residual signature are not to be generated (block810=no), then the flow proceeds to block 825. Therefore, as the flowleaves block 810 to proceed to block 825, some embodiments will havegenerated several pairs of detection signatures and residual signatures(e.g., a pair includes a detection signature and a residual signature),each corresponding to different detector spectral sensitivities ordifferent illuminant spectral-power distributions.

In block 825, respective detection signatures and residual signaturesfor candidate materials are obtained from a database. Next, in block830, a count j is initialized to 0. The flow then proceeds to block 835,where it is determined if the detection signatures and the residualsignatures of the image are to be compared with the detection signaturesand the residual signatures of another candidate material. If there aresignatures for J candidate materials, then if j<J (block 835=yes), acomparison is to be performed and the flow proceeds to block 840.

In block 840, the detection signatures of the image are compared to thedetection signatures of the current candidate material (the candidatematerial at index j). Next, in block 845 the residual signatures of theimage are compared to the residual signatures of the current candidatematerial. Also, in some embodiments, a specific pair of a detectionsignature and a residual signature of the image is compared with aspecific pair of a detection signature and a residual signature of acandidate material (e.g., pairs generated from similar detector spectralsensitivities or similar illuminant spectral-power distributions). Theflow then moves to block 850, where the count j is incremented, and thenthe flow returns to block 835.

If it is determined in block 835 that another comparison is not to beperformed (block 835=no), then the flow moves to block 855. In block855, a candidate material is selected for the image based on thecomparisons. Also, some embodiments of the method shown in FIG. 8 mayomit the detection signatures or, alternatively, the residualsignatures.

FIG. 9 illustrates an example embodiment of a system for identifying amaterial. The system includes a comparison device 910 and an imagestorage device 920. The comparison device 910 includes one or moreprocessors (CPU) 911, I/O interfaces 912, and storage/memory 913. TheCPU 911 includes one or more central processing units, which includemicroprocessors (e.g., a single core microprocessor, a multi-coremicroprocessor) and other circuits, and is configured to read andperform computer-executable instructions, such as instructions stored instorage or in memory (e.g., one or more modules that are stored instorage or memory). The computer-executable instructions may includethose for the performance of the methods described herein. The I/Ointerfaces 912 include communication interfaces to input and outputdevices, which may include a keyboard, a display, a mouse, a printingdevice, a touch screen, a light pen, an optical storage device, ascanner, a microphone, a camera, a drive, and a network (either wired orwireless).

The storage/memory 913 includes one or more computer-readable orcomputer-writable media, for example a computer-readable storage mediumor a transitory computer-readable medium. A computer-readable storagemedium is a tangible article of manufacture, for example, a magneticdisk (e.g., a floppy disk, a hard disk), an optical disc (e.g., a CD, aDVD, a Blu-ray), a magneto-optical disk, a magnetic tape, andsemiconductor memory (e.g., a non-volatile memory card, flash memory, asolid state drive, SRAM, DRAM, EPROM, EEPROM). A transitorycomputer-readable medium, for example a transitory propagating signal(e.g., a carrier wave), carries computer-readable information. Thestorage/memory 913 is configured to store computer-readable informationor computer-executable instructions, including, for example, detectorspectral sensitivities, illuminant spectral-power distributions,detection components, residual components, basis functions, detectionsignatures, residual signatures, and material signatures. The componentsof the comparison device 910 communicate via a bus.

The comparison device 910 also includes a component-calculation module914, a signature-generation module 915, and a comparison module 916. Insome embodiments, the comparison device 910 includes additional or fewermodules, the modules are combined into fewer modules, or the modules aredivided into more modules. The component-calculation module 914 includesinstructions that, when executed by the comparison device 910, cause thecomparison device 910 to obtain one or more images (e.g., from the imagestorage device 920), and calculate one or more of a detection componentand a residual component for the image. The signature-generation module915 includes instructions that, when executed by the comparison device910, cause the comparison device 910 to generate one or more detectionsignature, residual signature, or material signature for an image (i.e.,an object in the image). The comparison module 916 includes instructionsthat, when executed by the comparison device 910, cause the comparisondevice 910 to compare signatures (e.g., detection signatures, residualsignatures, or material signatures) and estimate a material based on thecomparison.

The image storage device 920 includes a CPU 922, storage/memory 923, I/Ointerfaces 924, and image storage 921. The image storage 921 includesone or more computer-readable media that are configured to store images.The image storage device 920 and the comparison device 910 communicatevia a network 990.

FIG. 10A illustrates an example embodiment of a system for identifying amaterial. The system includes an image storage device 1020, asignature-generation device 1010, and a diffusion-mapping device 1040,which communicate via a network 1090. The image storage device 1020includes one or more CPUs 1022, I/O interfaces 1024, storage/memory1023, and image storage 1021. The signature-generation device 1010includes one or more CPUs 1011, I/O interfaces 1012, storage/memory1014, and a signature-generation module 1013, which combines thefunctionality of the component-calculation module 914 and thesignature-generation module 915 of FIG. 9. The diffusion-mapping deviceincludes one or more CPUs 1041, I/O interfaces 1042, storage/memory1043, and a comparison module 1044.

FIG. 10B illustrates an example embodiment of a system for identifying amaterial. The system includes a comparison device 1050. The comparisondevice 1050 includes one or more CPUs 1051, I/O interfaces 1052,storage/memory 1053, an image storage module 1054, a spectral-propertystorage module 1055, a signature-generation module 1056, a comparisonmodule 1057, and a component-calculation module 1058. Thespectral-property storage module 1055 stores detector spectralsensitivities and illuminant spectral-power distributions. Thus, thisexample embodiment of the comparison device 1050 performs all theoperations and stores all the applicable information on a single device.

FIG. 11 illustrates an example embodiment of a candidate materialdataset, which may be stored in a database. The original textiles arespecified in the dataset. Six different dyes (Dark Blue (Navy Blue),Dark Green (Dark Green), Red (Scarlet), Light Blue (Royal Blue), Purple(Purple), and Yellow (Lemon Yellow)) were used to dye the samples, andthere are 7 samples per textile, considering the un-dyed originalsamples. Also, the materials are numbered 1-10, and these numberings areconsistent from FIG. 11 to FIG. 15. The spectral reflectance of eachsample material was measured, and the results are shown in FIG. 12,which illustrates an example of spectral reflectances of materials in acandidate material dataset. A correlation analysis was performed for anunknown material (material 10) on the materials in the dataset for all36 channels of sampling, and the correlation results are presented inFIG. 13, which illustrates examples of spectral reflectance correlationsbetween a material and materials in a candidate material dataset. Theaverage correlation results were generated across all 7 samples,including the original material and the material dyed with 6 differentcolors. In a row in FIG. 13, the top value is the minimum correlation,the middle value is the average correlation, and the bottom value is themaximum correlation.

According to the average correlation results, the most likely materialcomposition of the unknown material (material 10) is material 3, asindicated by a correlation of 0.919 based on average values. This isconsistent with a visual comparison of the plots of material 10 andmaterial 3. However, by just examining spectral correlation,differentiating most of the other materials may be difficult because allthe materials have a relatively high average spectral correlation withthe unknown material (material 10).

Thus, in addition to the overall spectral correlation analysis, acorrelation analysis using decomposition was performed that includedgenerating correlation coefficients for detection component coefficientsand residual component coefficients. The residual component coefficientswere metameric black component coefficients. The detection componentcorrelation results are shown in FIG. 14, which illustrates examples ofspectral reflectance correlations between a material and materials in acandidate material dataset. The detection components were calculatedusing three channels (e.g., for the fundamental components case of thedetection components uses three channels by definition). In a row, thetop value is the minimum correlation, the middle value is the averagecorrelation, and the bottom value is the maximum correlation. FIG. 14shows that all of the detection component correlation coefficients werehigh, which means that they all have same color. This result wasexpected since the same dye was used for each set of materials.

Additionally, the correlation coefficients for the residual components(metameric black components in this example) are shown in FIG. 15, whichillustrates examples of spectral reflectance correlations between amaterial and materials in a candidate material dataset. Thirty-sixchannels were used to calculate the residual components. FIG. 15 showsthat the correlation coefficients for the residual components werelower, which shows that the materials are not the same in mostinstances. However, the average correlations between material 6 andmaterial 9 were relatively high (0.664) since they are very similar, asshown in the curves of FIG. 12.

The cross-correlation values in FIG. 15 show the potentialdiscrimination of each material in relation to other materials. Forexample, by establishing a correlation of 0.6 as a threshold fordiscrimination, the discrimination success rate may be calculatedaccording to the following: (number of samples with correlation <0.6)divided by (total number of samples). If this discrimination successrate is applied to the results for materials 1-10, as shown in FIG. 13to FIG. 15, the overall spectral correlation produces an averagediscrimination success rate of 5.7%, which can be improved by 78.3%through using metameric black correlation. For correlations below 0.6,the discrimination success rates for both the overall spectralcorrelations and the residual component correlations are shown in FIG.16.

FIG. 17 illustrates an example embodiment of a candidate materialdataset, which may be stored in a database. The dataset includescategories of natural and artificial (“man-made”) objects. The spectralreflectances of 192 material samples (88 natural material samples and104 man-made material samples) from the illustrated categories weremeasured, and the results are shown in FIG. 18, which illustrates anexample of spectral reflectances of materials in a candidate materialdataset. Also, FIG. 19A shows the average overall spectral correlationsfor 88 natural samples and 104 man-made samples for different numbers ofchannels.

Furthermore, based on CIE 2 degree observer and D65 illuminant, thematerial samples were decomposed into respective fundamental componentsand metameric black components before calculating the correlation. FIG.19B shows the average correlations of the material samples that werecalculated based on the fundamental components. The natural materialsamples had slightly better discrimination performance than the wholeensemble of material samples and the man-made material samples. Also,some embodiments that use the fundamental component for the detectioncomponent do not use more than three channels because by definition thefundamental component has three channels.

FIG. 19C shows the average correlations of the material samples thatwere calculated based on metameric black components. FIG. 19C shows thatthe metameric black component analysis has a much better discriminationperformance than the fundamental component analysis.

FIG. 20A shows a comparison between the overall spectral correlationresults and the metameric black correlation results for the 88 naturalsamples. FIG. 20B shows the improvement of the metameric blackcorrelation results over the overall spectral correlation results forthe 88 natural samples in terms of the number of channels. The resultsin FIG. 20B show that for the natural material samples there is someimprovement in performance using metameric black component correlation:it was possible to achieve improvements between 15% and 28% using morethan 3 channels.

FIG. 21A shows a comparison between the results of overall spectralcorrelation and metameric black correlation for the 104 man-madematerial samples. FIG. 21B shows the improvement of the metameric blackcorrelation results over the overall spectral correlation results forthe 104 man-made material samples in terms of number of channels. Theresults in FIG. 21B show the great benefit of using metameric blackcorrelation instead of spectral correlation for these man-madematerials, with the results showing improvements of over 30% when using5 or more channels. This dramatic improvement is obtained because bydecomposing the spectra into fundamental and metameric black, the“color” component (related to the fundamental component) can bedecoupled from a residual component. By analyzing the residual component(e.g., the metameric black component) the effects of colors that are notnatural in the properties of a man-made material may be discounted.

FIG. 22 shows the discrimination success rates for examples of overallspectral correlations and residual component correlations. Thediscrimination success rates were generated using 6 channels for 15representative samples that included both natural and man-madematerials. The results show that using metameric black correlation mayimprove the discrimination rate for man-made objects, such as paints andplastics.

FIG. 23A illustrates a graphic representation of a transformation matrix[T]. In the transformation matrix [T], the z-axis shows the value andthe x- and y-axes show the wavelength numbers. The 3 peaks correspond tothe peaks of the CIE 2 degree color matching functions combined with thespectral power distribution of CIE D65 illuminant. FIG. 23B shows anexample of detection components. The example detection components (e.g.,fundamental component) were calculated from the previous example. Theaxis on the right side represents wavelength number and the axis on theleft side represents sample number. FIG. 23C shows an example ofresidual components. The example residual components (e.g., metamericblack component) were calculated from the previous example. In FIG. 23C,the axis on the right side represents wavelength number and the axis onthe left side represents sample number.

The above-described devices, systems, and methods can be implemented bysupplying one or more computer-readable media that containcomputer-executable instructions for realizing the above-describedoperations to one or more computing devices that are configured to readand execute the computer-executable instructions. Thus, the systems ordevices perform the operations of the above-described embodiments whenexecuting the computer-executable instructions. Also, an operatingsystem on the one or more systems or devices may implement at least someof the operations of the above-described embodiments. Thus, thecomputer-executable instructions or the one or more computer-readablemedia that contain the computer-executable instructions constitute anembodiment.

Any applicable computer-readable medium (e.g., a magnetic disk(including a floppy disk, a hard disk), an optical disc (including a CD,a DVD, a Blu-ray disc), a magneto-optical disk, a magnetic tape, andsemiconductor memory (including flash memory, DRAM, SRAM, a solid statedrive, EPROM, EEPROM)) can be employed as a computer-readable medium forthe computer-executable instructions. The computer-executableinstructions may be stored on a computer-readable storage medium that isprovided on a function-extension board inserted into a device or on afunction-extension unit connected to the device, and a CPU provided onthe function-extension board or unit may implement at least some of theoperations of the above-described embodiments.

The scope of the claims is not limited to the above-describedembodiments and includes various modifications and equivalentarrangements. Also, as used herein, the conjunction “or” generallyrefers to an inclusive “or,” though “or” may refer to an exclusive “or”if expressly indicated or if the context indicates that the “or” must bean exclusive “or.”

What is claimed is:
 1. A method comprising: obtaining an image;estimating a spectral image of the obtained image; calculating one orboth of a detection component of the image and a residual component ofthe image, wherein the detection component of the image is calculatedbased on the spectral image, a spectral-power distribution for aspecific illuminant, and spectral sensitivities of a detector, andwherein the residual component of the image is calculated based on thespectral image, the spectral power distribution for the specificilluminant, and the spectral sensitivities of the detector or iscalculated based on the spectral image and the calculated detectioncomponent; and generating an image signature based on one or both of thedetection component and detection-component basis functions, and theresidual component and residual-component basis functions.
 2. The methodof claim 1, further comprising identifying one or more materials in theimage based on the image signature.
 3. The method of claim 1, whereingenerating the image signature based on the detection component and thedetection-component basis functions includes calculating coefficients ofthe detection-component basis functions based on the detection componentand on the detection-component basis functions, and wherein generatingthe image signature based on the residual component and theresidual-component basis functions includes calculating coefficients ofthe residual-component basis functions based on the residual componentand on the residual-component basis functions.
 4. The method of claim 3,wherein calculating the coefficients of the detection-component basisfunctions includes decomposing the detection component into thedetection-component basis functions; and wherein calculating thecoefficients of the residual-component basis functions includesdecomposing the residual component into the residual-component basisfunctions.
 5. The method of claim 3, further comprising: generating acolorimetric signature based on the coefficients of thedetection-component basis functions; and generating a spectral signaturebased on the coefficients of the residual-component basis functions,wherein the image signature is generated based on the colorimetricsignature and the spectral signature.
 6. The method of claim 1, whereinboth the detection component of the image and the residual component ofthe image are calculated, and wherein the image signature is generatedbased on both of the detection component and the detection-componentbasis functions, and the residual component and the residual-componentbasis functions.
 7. The method of claim 6, wherein generating the imagesignature includes calculating both of coefficients of thedetection-component basis functions and coefficients of theresidual-component basis functions.
 8. The method of claim 1, whereinthe detection component is a fundamental component and wherein theresidual component is a metameric black component.
 9. A system forgenerating a spectral signature, the system comprising: one or morecomputer-readable media; and one or more processors configured to causethe system to perform operations including obtaining an image,generating a spectral reflectance estimate of the image, calculating oneor both of a detection component and a residual component, wherein thedetection component is calculated based on the spectral reflectanceestimate of the image, a spectral-power distribution for a specificilluminant, and detector spectral sensitivities, and wherein theresidual component is calculated based on the spectral reflectanceestimate of the image and the calculated detection component or iscalculated based on the spectral reflectance estimate of the image, thespectral power distribution for the specific illuminant, and thedetector spectral sensitivities, and generating one or both of adetection-component signature and a residual-component signature,wherein the detection component signature is generated based on thedetection component and on detection-component basis functions, andwherein the residual-component signature is generated based on theresidual component and on residual-component basis functions.
 10. Thesystem of claim 9, wherein, to generate the detection-componentsignature, the one or more processors are further configured to causethe system to calculate coefficients of the detection-component basisfunctions based on the detection component and on thedetection-component basis functions, and wherein, to generate theresidual-component signature, the one or more processors are furtherconfigured to cause the system to calculate coefficients of theresidual-component basis functions based on the residual component andon the residual-component basis functions.
 11. The system of claim 10,wherein, to calculate the coefficients of the detection-component basisfunctions, the one or more processors are further configured to causethe system to decompose the detection component into thedetection-component basis functions, and wherein, to calculate thecoefficients of the residual-component basis functions, the one or moreprocessors are further configured to cause the system to decompose theresidual component into the residual-component basis functions.
 12. Thesystem of claim 11, wherein the one or more processors are furtherconfigured to cause the system to perform operations comprising:obtaining one or both of respective detection-component signatures andrespective residual-component signatures for one or more candidatematerials; and selecting a candidate material for the image based on thedetection-component signature for the image and the detection-componentsignatures for the candidate images, on the residual-component signaturefor the image and the residual-component signatures for the candidateimages, or on both.
 13. One or more non-transitory computer-readablemedia storing instructions that, when executed by one or more computingdevices, cause the one or more computing devices to perform operationscomprising: obtaining an image; generating a spectral image of theobtained image; calculating one or more of a detection component and aresidual component, wherein the detection component is calculated basedon the spectral image, a spectral-power distribution for a specificilluminant, and detector spectral sensitivities, and wherein theresidual component is calculated based on the spectral image and thecalculated detection component or is calculated based on the spectralimage, the spectral power distribution for the specific illuminant, andthe detector spectral sensitivities; and generating an image signaturebased on at least one of the detection component and detection-componentbasis functions, and the residual component and residual-component basisfunctions.
 14. The one or more non-transitory computer-readable media ofclaim 13, wherein the operations include calculating both the detectioncomponent and the residual component.
 15. The one or more non-transitorycomputer-readable media of claim 13, wherein generating the imagesignature based on the detection component and the detection-componentbasis functions includes calculating coefficients of thedetection-component basis functions based on the detection component andon the detection-component basis functions, and wherein generating theimage signature based on the residual component and theresidual-component basis functions includes calculating coefficients ofthe residual-component basis functions based on the residual componentand on the residual-component basis functions.
 16. The one or morenon-transitory computer-readable media of claim 15, wherein generatingthe image signature includes one or both of generating a colorimetricsignature based on the coefficients of the detection-component basisfunctions and generating a spectral signature based on the coefficientsof the residual-component basis functions.
 17. The one or morenon-transitory computer-readable media of claim 16, wherein theoperations further comprise comparing the spectral signature to acollection of corresponding spectral signatures of candidate materials,and identifying a material in the image based on the comparisons. 18.The one or more non-transitory computer-readable media of claim 17,wherein comparing the spectral signature to a collection ofcorresponding spectral signatures of candidate materials is based on aspectral matching probability function.
 19. The one or morenon-transitory computer-readable media of claim 15, wherein calculatingthe coefficients of the residual-component basis functions includesdecomposing the residual component into the residual-component basisfunctions.
 20. The one or more non-transitory computer-readable media ofclaim 19, wherein calculating the coefficients of thedetection-component basis functions includes decomposing the detectioncomponent into the detection-component basis functions.